Systems and methods for generating biomarker signatures with integrated bias correction and class prediction

ABSTRACT

Described herein are systems and methods for correcting a data set and classifying the data set in an integrated manner. A training data set, a training class set, and a test data set are received. A first classifier is generated for the training data set by applying a machine learning technique to the training data set and the training class set, and a first test class set is generated by classifying the elements in the test data set according to the first classifier. For each of multiple iterations, the training data set is transformed, the test data set is transformed, and a second classifier is generated by applying a machine learning technique to the transformed training data set. A second test class set is generated according to the second classifier, and the first test class set is compared to the second test class set.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. Nationalization of PCT Application No.PCT/EP2013/062980 filed Jun. 21, 2013, which claims the benefit of U.S.Provisional Application No. 61/662,792, filed Jun. 21, 2012, theentireties of which are incorporated herein by reference.

BACKGROUND

In the biomedical field it is important to identify substances that areindicative of a specific biological state, namely biomarkers. As newtechnologies of genomics and proteomics emerge, biomarkers are becomingmore and more important in biological discovery, drug development andhealth care. Biomarkers are not only useful for diagnosis and prognosisof many diseases, but also for understanding the basis for developmentof therapeutics. Successful and effective identification of biomarkerscan accelerate the new drug development process. With the combination oftherapeutics with diagnostics and prognosis, biomarker identificationwill also enhance the quality of current medical treatments, thus playan important role in the use of pharmacogenetics, pharmacogenomics andpharmacoproteomics.

Genomic and proteomic analysis, including high throughput screening,supplies a wealth of information regarding the numbers and forms ofproteins expressed in a cell and provides the potential to identify foreach cell, a profile of expressed proteins characteristic of aparticular cell state. In certain cases, this cell state may becharacteristic of an abnormal physiological response associated with adisease. Consequently, identifying and comparing a cell state from apatient with a disease to that of a corresponding cell from a normalpatient can provide opportunities to diagnose and treat diseases.

These high throughput screening techniques provide large data sets ofgene expression information. Researchers have attempted to developmethods for organizing these data sets into patterns that arereproducibly diagnostic for diverse populations of individuals. Oneapproach has been to pool data from multiple sources to form a combineddata set and then to divide the data set into a discovery/training setand a test/validation set. However, both transcription profiling dataand protein expression profiling data are often characterized by a largenumber of variables relative to the available number of samples.

Observed differences between expression profiles of specimens fromgroups of patients or controls are typically overshadowed by severalfactors, including biological variability or unknown sub-phenotypeswithin the disease or control populations, site-specific biases due todifference in study protocols, specimens handling, biases due todifferences in instrument conditions (e.g., chip batches, etc), andvariations due to measurement error. Some techniques attempt to correctto for bias in the data samples (which may result from, for example,having more of one class of sample represented in the data set thananother class).

Several computer-based methods have been developed to find a set offeatures (markers) that best explain the difference between the diseaseand control samples. Some early methods included statistical tests suchas LIMMA, the FDA approved mammaprint technique for identifyingbiomarkers relating to breast cancer, logistical regression techniquesand machine learning methods such as support vector machines (SVM).Generally, from a machine learning perspective, the selection ofbiomarkers is typically a feature selection problem for a classificationtask. However, these early solutions faced several disadvantages. Thesignatures generated by these techniques were often not reproduciblebecause the inclusion and exclusion of subjects can lead to differentsignatures. These early solutions also generated many false positivesignatures and were not robust because they operated on datasets havingsmall sample sizes and high dimensions.

Accordingly there is a need for improved techniques for identifyingbiomarkers for clinical diagnosis and/or prognosis, and more generally,for identifying data markers that can be used to classify elements in adata set into two or more classes.

SUMMARY

Applicants have recognized that existing computer-based methodsdisadvantageously apply bias correction techniques separately from classprediction techniques. The computer systems and computer programproducts described herein implement methods that apply an integratedapproach to bias correction and class prediction, which may achieveimproved classification performance in biomarker and other dataclassification applications. In particular, the computer-implementedmethods disclosed herein adopt an iterative approach to bias correctionand class prediction. In various embodiments of the computer-implementedmethods, at least one processor in the system receives a training dataset and a training class set, the training class set identifying a classassociated with each of the elements in the training data set. Theprocessor in the system also receives a test data set. The processorgenerates a first classifier for the training data set by applying amachine learning technique to the training data set and the trainingclass set, and generates a first test class set by classifying theelements in the test data set according to the first classifier. Foreach of multiple iterations, the processor: transforms the training dataset based on at least one of the training class set and the test classset, transforms the test data set by applying the transformation of theprevious step, generates a second classifier for the transformedtraining data set by applying a machine learning technique to thetransformed training data set and the training class set, and generatesa second test class set by classifying the elements in the transformedtest data set according to the second classifier. The processor alsocompares the first test class set to the second test class set, and whenthe first test class set and the second test class set differ, theprocessor stores the second class set as the first class set, stores thetransformed test data set as the test data set and returns to thebeginning of the iteration. The computer systems of the inventioncomprises means for implementing the methods and its various embodimentsas described above.

In certain embodiments of the methods described above, the methodfurther comprises outputting the second class set when the first testclass set and the second test class set do not differ. In particular,the iterations as described above may be repeated until the first testclass set and the second test class set converge, and there is nodifference between the predicted classifications. In certain embodimentsof the methods described above, an element of the training data setrepresents gene expression data for a patient with a disease, for apatient resistant to the disease, or for a patient without the disease.The elements of the training class set may correspond to known classidentifiers for the data samples in the training data set. For example,the class identifiers may include categories such as “Disease Positive,”“Disease Immune,” or “Disease Free.”

In certain embodiments of the methods described above, the training dataset and the test data set are generated by randomly assigning samples inan aggregate data set to the training data set or the test data set.Randomly splitting the aggregate data set into the training data set andthe test data set may be desirable for predicting classes and generatingrobust gene signatures. Furthermore, samples of the aggregate data setmay be discarded prior to the splitting, or samples of the training dataset or the test data set may be discarded after the splitting. Incertain embodiments of the methods described above, the step oftransforming the training data set, transforming the test data set, orboth steps of transforming the training data set and transforming thetest data set comprise performing a bias correction technique byadjusting the elements of the data set based on a centroid of the dataset. The transformation is performed according to a transformationfunction, which may define the transformation based on the trainingclass set. In certain embodiments of the methods described above, thethe bias correction technique comprises subtracting a component of thecentroid from each element of the data set. For example, the result ofthe bias correction technique may be that each element of the trainingdata set, the test data set, or both the training and test data sets is“recentered” by taking into account the centroids of each classrepresented in the data set. In certain embodiments of the methodsdescribed above, the step of transforming the training data set,transforming the test data set, or both steps of transforming thetraining data set and transforming the test data set comprise applying arotation, a shear, a shift, a linear transformation, or a non-lineartransformation.

In certain embodiments of the methods described above, the methodsfurther comprise comparing the first test class set to the second testclass set for each of the plurality of iterations. As a result of thecomparison, the first test class set and the second test class set maybe said to differ if any single element of the first test class setdiffers from a corresponding element of the second test class set. Ingeneral, a threshold may be set such that the first test class set andthe second test class set are said to differ if at least a predeterminednumber of elements in the first test class set differs from thecorresponding elements in the second test class set.

In certain embodiments of the methods described above, the methodsfurther comprise generating the second classifier for the transformedtraining data set by applying a machine learning technique to thetransformed training data set and the training class set for each of theplurality of iterations. In certain embodiments of the methods describedabove, the transforming of the test data set involves the sametransformation as the transformation of the transforming of the trainingdata set. In certain embodiments of the methods described above, themethods further comprise providing the second test class set to adisplay device, a printing device, or a storing device. In certainembodiments of the methods described above, the methods further comprisecomputing a performance metric of the second classifier based on anerror rate. In certain embodiments, linear classifiers such as but notlimited to Linear Discriminant Analysis (LDA), logistic regression,support vector machine, naive Bayes classifier, are preferred.

The computer systems of the present invention comprise means forimplementing the various embodiments of the methods, as described above.For example, a computer program product is described, the productcomprising computer-readable instructions that, when executed in acomputerized system comprising at least one processor, cause theprocessor to carry out one or more steps of any of the methods describedabove. In another example, a computerized system is described, thesystem comprising a processor configured with non-transitorycomputer-readable instructions that, when executed, cause the processorto carry out any of the methods described above. The computer programproduct and the computerized methods described herein may be implementedin a computerized system having one or more computing devices, eachincluding one or more processors. Generally, the computerized systemsdescribed herein may comprise one or more engines, which include aprocessor or devices, such as a computer, microprocessor, logic deviceor other device or processor that is configured with hardware, firmware,and software to carry out one or more of the computerized methodsdescribed herein. Any one or more of these engines may be physicallyseparable from any one or more other engines, or may include multiplephysically separable components, such as separate processors on commonor different circuit boards. The computer systems of the presentinvention comprises means for implementing the methods and its variousembodiments as described above. The engines may be interconnected fromtime to time, and further connected from time to time to one or moredatabases, including a perturbations database, a measurables database,an experimental data database and a literature database. Thecomputerized system described herein may include a distributedcomputerized system having one or more processors and engines thatcommunicate through a network interface. Such an implementation may beappropriate for distributed computing over multiple communicationsystems.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features of the disclosure, its nature and various advantages,will be apparent upon consideration of the following detaileddescription, taken in conjunction with the accompanying drawings, inwhich like reference characters refer to like parts throughout, and inwhich:

FIG. 1 depicts an exemplary system for identifying one or more biomarkersignatures;

FIG. 2 illustrates the classification of elements in a data set;

FIGS. 3A and 3B are flow diagrams of an exemplary process forclassifying a data set;

FIG. 4 is a block diagram of a computing device, such as any of thecomponents of the system of FIG. 1;

FIG. 5 is a heatmap of a gene signature in a training data set.

DETAILED DESCRIPTION

To provide an overall understanding of the systems and methods describedherein, certain illustrative embodiments will now be described,including systems and methods for identifying gene biomarker signatures.However, it will be understood by one of ordinary skill in the art thatthe systems, computer program products and methods described herein maybe adapted and modified for other suitable applications, such as anydata classification application, and that such other additions andmodifications will not depart from the scope thereof. Generally, thecomputerized systems described herein may comprise one or more engines,processor or devices, such as a computer, microprocessor, or logicdevice that is configured with hardware, firmware, and software to carryout one or more of the computerized methods described herein.

FIG. 1 depicts an exemplary system 100 for identifying one or morebiomarker signatures in which the classification techniques disclosedherein may be implemented. The system 100 includes a biomarker generator102 and a biomarker consolidator 104. The system 100 further includes acentral control unit (CCU) 101 for controlling certain aspects of theoperation of the biomarker generator 102 and the biomarker consolidator104. During operation, data such as gene expression data is received atthe biomarker generator 102. The biomarker generator 102 processes thedata to generate a plurality of candidate biomarkers and correspondingerror rates. The biomarker consolidator 104 receives these candidatebiomarkers and error rates and selects a suitable biomarker having anoptimal performance measure and size.

The biomarker generator 102 includes several components for processingdata and generating a set of candidate biomarkers and candidate errorrates. In particular, the biomarker generator includes a datapre-processing engine 110 for splitting the data into a training dataset and a test data set. The biomarker generator 102 includes aclassification engine 114 for receiving the training data set and thetest data set and classifying the elements of the test data set into oneof two or more classes (e.g., diseased and non-diseased, susceptible andimmune and diseased, etc.). The biomarker generator 102 includes aclassifier performance monitoring engine 116 for determining theperformance of the classifier as applied to the test data selected bythe data pre-processing engine 110. The classifier performancemonitoring engine 116 identifies candidate biomarkers based on theclassifier (e.g., the components of the elements of the data set thatare most important to the classification) and generates performancemeasures, which may include candidate error rates, for one or morecandidate biomarkers. The biomarker generator 102 further includes abiomarker store 118 for storing one or more candidate biomarkers andcandidate performance measures.

The biomarker generator may be controlled by the CCU 101, which in turnmay be automatically controlled or user-operated. In certainembodiments, the biomarker generator 102 may operate to generate aplurality of candidate biomarkers, each time splitting the data randomlyinto training and test data sets. To generate such a plurality ofcandidate biomarkers, the operation of the biomarker generator 102 maybe iterated a plurality of times. CCU 101 may receive one or more systemiteration parameters including a desired number of candidate biomarkers,which in turn may be used to determine the number of times the operationof the biomarker generator 102 may be iterated. The CCU 101 may alsoreceive other system parameters including a desired biomarker size whichmay be representative of the number of components in a biomarker (e.g.,the number of genes in a biomarker gene signature). The biomarker sizeinformation may be used by the classifier performance monitoring engine116 for generating candidate biomarkers from the training data. Theoperation of the biomarker generator 102, and the classification engine114 in particular, are described in more detail with reference to FIGS.2-4.

The biomarker generator 102 generates one or more candidate biomarkersand candidate error rates, which is used by the biomarker consolidator104 for generating robust biomarkers. The biomarker consolidator 104includes a biomarker consensus engine 128 which receives a plurality ofcandidate biomarkers and generates a new biomarker signature having themost frequently occurring genes across the plurality of candidatebiomarkers. The biomarker consolidator 104 includes an error calculationengine 130 for determining an overall error rate across the plurality ofcandidate biomarkers. Similar to the biomarker generator 102, thebiomarker consolidator 104 may also be controlled by the CCU 101, whichin turn may be automatically controlled or user-operated. The CCU 101may receive and/or determine suitable threshold values for the minimumbiomarker size, and use this information to determine the number ofiterations to operate both the biomarker generator 102 and the biomarkerconsolidator 104. In one embodiment, during each iteration, the CCU 101decreases the biomarker size by one and iterates both the biomarkergenerator 102 and the biomarker consolidator 104 until the threshold isreached. In such an embodiment, the biomarker consensus engine 128outputs a new biomarker signature and a new overall error rate for eachiteration. The biomarker consensus engine 128 thus outputs set of newbiomarker signatures each having a different size varying from thethreshold value up to a maximum biomarker size. The biomarkerconsolidator 104 further includes a biomarker selection engine 126 whichreviews the performance measure or error rate of each of these newbiomarker signatures and selects the optimal biomarker for output. Theoperation of the biomarker consolidator 104 and its respective enginesare described in more detail with reference to FIGS. 2-4.

FIGS. 3A and 3B are flow diagrams of an exemplary process forclassifying a data set. At step 302, the classification engine 114receives training data and test data. As described below, theclassification engine 114 uses the training data to develop one or moreclassifiers, then applies the one or more classifiers to the test data.As illustrated in FIGS. 3A and 3B, the training data includes a trainingdata set T0.train 304 and a training class set cl.train 306. Eachelement in the training data set T0.train 304 represents a data sample(e.g., a vector of expression data from a particular patient) andcorresponds to a known class identifier in the training class setc1.train 306. For example, in a three-class scenario, the first elementin the training data set T0.train 304 may represent gene expression datafor a patient with a particular disease, and may correspond to a firstelement “Disease Positive” in the training class set c1.train 306; thesecond element in the training data set T0.train 304 may represent geneexpression data for a patient who is resistant to or immune to theparticular disease, and may correspond to a second element “DiseaseImmune” in the training class set c1.train 306; and the third element inthe training data set T0.train 304 may represent gene expression datafor a patient without the particular disease, and may correspond to athird element “Disease Free” in the training class set c1.train 306. Thetest data received at step 302 includes the test data set T0.test 308,which represents the same underlying type of data as the data samples inthe training data set T0.train 304, but may represent samples taken fromdifferent patients or different experiments, for example. Optionally,the classification engine 114 also receives a test class set c1.test 310that includes the known class identifiers for the data samples in thetest data set, which may be used to evaluate the performance of theclassifier generated by the classification engine 114 when thatclassifier is applied to the test data set T0.test 308. In someimplementations, no known classes for the data samples in the test dataset T0.test 308 are available, and thus the test class set c1.test 310is not provided to the classification engine 114.

Generally, the data received at step 302 may represent any experimentalor otherwise obtained data from which a classification may be drawn,such as expression values of a plurality of different genes in a sample,and/or a variety of a phenotypic characteristics such as levels of anybiologically significant analyte. In certain embodiments, the data setsmay include expression level data for a disease condition and for acontrol condition. As used herein, the term “gene expression level” mayrefer to the amount of a molecule encoded by the gene, e.g., an RNA orpolypeptide. The expression level of an mRNA molecule may include theamount of mRNA (which is determined by the transcriptional activity ofthe gene encoding the mRNA) and the stability of the mRNA (which isdetermined by the half-life of the mRNA). The gene expression level mayalso include the amount of a polypeptide corresponding to a given aminoacid sequence encoded by a gene. Accordingly, the expression level of agene can correspond to the amount of mRNA transcribed from the gene, theamount of polypeptide encoded by the gene, or both. Expression levels ofa gene may be further categorized by expression levels of differentforms of gene products. For example, RNA molecules encoded by a gene mayinclude differentially expressed splice variants, transcripts havingdifferent start or stop sites, and/or other differentially processedforms. Polypeptides encoded by a gene may encompass cleaved and/ormodified forms of polypeptides. Polypeptides can be modified byphosphorylation, lipidation, prenylation, sulfation, hydroxylation,acetylation, ribosylation, farnesylation, addition of carbohydrates, andthe like. Further, multiple forms of a polypeptide having a given typeof modification can exist. For example, a polypeptide may bephosphorylated at multiple sites and express different levels ofdifferentially phosphorylated proteins.

In certain embodiments the gene expression level in a cell or tissue maybe represented by a gene expression profile. Gene expression profilesmay refers to a characteristic representation of a gene's expressionlevel in a specimen such as a cell or tissue. The determination of agene expression profile in a specimen from an individual isrepresentative of the gene expression state of the individual. A geneexpression profile reflects the expression of messenger RNA orpolypeptide or a form thereof encoded by one or more genes in a cell ortissue. An expression profile may generally refer to a profile ofbiomolecules (nucleic acids, proteins, carbohydrates) which showsdifferent expression patterns among different cells or tissue. A datasample representing a gene expression profile may be stored as a vectorof expression levels, with each entry in the vector corresponding to aparticular biomolecule or other biological entity.

In certain embodiments, the data sets may include elements representinggene expression values of a plurality of different genes in a sample. Inother embodiments, the data set may include elements that representpeaks detected by mass spectrometry. Generally, each data set mayinclude data samples that each correspond to one of a plurality ofbiological state classes. For example, a biological state class caninclude, but is not limited to: presence/absense of a disease in thesource of the sample (i.e., a patient from whom the sample is obtained);stage of a disease; risk for a disease; likelihood of recurrence ofdisease; a shared genotype at one or more genetic loci (e.g., a commonHLA haplotype; a mutation in a gene; modification of a gene, such asmethylation, etc.); exposure to an agent (e.g., such as a toxicsubstance or a potentially toxic substance, an environmental pollutant,a candidate drug, etc.) or condition (temperature, pH, etc); ademographic characteristic (age, gender, weight; family history; historyof preexisting conditions, etc.); resistance to agent, sensitivity to anagent (e.g., responsiveness to a drug) and the like.

Data sets may be independent of each other to reduce collection bias inultimate classifier selection. For example, they can be collected frommultiple sources and may be collected at different times and fromdifferent locations using different exclusion or inclusion criteria,i.e., the data sets may be relatively heterogeneous when consideringcharacteristics outside of the characteristic defining the biologicalstate class. Factors contributing to heterogeneity include, but are notlimited to, biological variability due to sex, age, ethnicity;individual variability due to eating, exercise, sleeping behavior; andsample handling variability due to clinical protocols for bloodprocessing. However, a biological state class may comprise one or morecommon characteristics (e.g., the sample sources may representindividuals having a disease and the same gender or one or more othercommon demographic characteristics). In certain embodiments, the datasets from multiple sources are generated by collection of samples fromthe same population of patients at different times and/or underdifferent conditions.

In certain embodiments, a plurality of data sets is obtained from aplurality of different clinical trial sites and each data set comprisesa plurality of patient samples obtained at each individual trial site.Sample types include, but are not limited to, blood, serum, plasma,nipple aspirate, urine, tears, saliva, spinal fluid, lymph, cell and/ortissue lysates, laser microdissected tissue or cell samples, embeddedcells or tissues (e.g., in paraffin blocks or frozen); fresh or archivalsamples (e.g., from autopsies). A sample can be derived, for example,from cell or tissue cultures in vitro. Alternatively, a sample can bederived from a living organism or from a population of organisms, suchas single-celled organisms. In one example, when identifying biomarkersfor a particular cancer, blood samples for might be collected fromsubjects selected by independent groups at two different test sites,thereby providing the samples from which the independent data sets willbe developed.

In some implementations, the training and test sets are generated by thedata pre-processing engine 110 (FIG. 1), which receives bulk data andsplits the bulk data into a training data set and a test data set. Incertain embodiments, the data pre-processing engine 110 randomly splitsthe data into these two groups. Randomly splitting the data may bedesirable for predicting classes and generating robust gene signature.In other embodiments, the data pre-processing engine 110 splits the datainto two or more groups based on the type or label of the data.Generally, the data can be split into a training data set and a testdata set in any suitable way as desired without departing from the scopeof the present disclosure. The training data set and the test data setmay have any suitable size and may be of the same or different sizes. Incertain embodiments, the data pre-processing engine 110 may discard oneor more pieces of data prior to splitting the data into the training andtest data sets. In certain embodiments, the data pre-processing engine110 may discard one or more pieces of data from the training data setand/or the test data set prior to any further processing.

At step 311, the classification engine 114 sets a counter variable iequal to 1. At step 312, the classification engine 114 generates a firstclassifier rf 314 based on the training data set T0.train 304 and thetraining class set c1.train 306. FIG. 2 illustrates the classificationof elements in a data set. The classification engine 114 may use any oneor more known machine-learning algorithms at step 312, including but notlimited to support vector machine techniques, linear discriminantanalysis techniques, Random Forest techniques, k-nearest neighborstechniques, partial least squares techniques (including techniques thatcombine partial least squares and linear discriminant analysisfeatures), logistic regression techniques, neural network-basedtechniques, decision tree-based techniques and shrunken centroidtechniques (e.g., as described by Tibshirani, Hastle, Narasimhan and Chuin “Diagnosis of multiple cancer types by shrunken centroids of geneexpression,” PNAS, v. 99, n. 10, 2002). A number of such techniques areavailable as packages for the R programming language, including lda,svm, randomForest, knn, pls.lda and pamr, corresponding to lineardiscriminant analysis, support vector machine, random forest (Breiman,Machine Learning, 45(1):5-32 (2001)), k-nearest neighbors (Bishop,Neural Networks for Pattern Recognition, ed. O.U. Press, 1995), partialleast squares discriminant analysis, and PAMR (Tibshirani et al., ProcNatl Acad Sci USA, 99(10):6567-6572 (2002)). The classification engine114 may store the first classifier rf 314 in a memory at step 312.

At step 316, the classification engine 114 generates a set of predictedtest classifications predc1.test 318 by applying the first classifier rf314 (generated at step 312) to the test data set T0.test 308. Theclassification engine 114 may store the predicted classificationspredc1.test 318 in a memory at step 316.

At step 320, the classification engine 114 transforms the training dataset T0.train 304. This transformation proceeds according to atransformation function, correctedData, which transforms the trainingdata set T0.train 304 based on the training class set c1.train 306. Theresult of the transformation of step 310 is a transformed training dataset, T0.train.2 322, which the classification engine 114 may store in amemory. In some implementations, the transformation performed by theclassification engine 114 at step 320 includes a bias correctiontechnique. For example, the transformation may “recenter” the trainingdata set T0.train 304 by adjusting the elements of the training data setT0.train 304 with respect to the centroid of the data set taken as awhole, or the centroids of each class represented in the data set.

One particular recentering technique involves centering the elements ofthe training data set T0.train 304 based on the center of centroids ofdifferent groups. If there are n data samples in the training data setT0.train 304, and each data sample is a vector with p entries (e.g.,representing expression levels for p different genes), let xij representthe ith entry of data sample j. If the training class set c1.train 308represents K different classes, let Ck represent the indices of the nksamples in class k. The classification engine 114 may calculate the ithcomponent of the centroid of class k as

$\begin{matrix}{{\overset{\_}{x}}_{ik} = {\sum\limits_{j \in C_{k}}\;{\frac{x_{ij}}{n_{k}}.}}} & (1)\end{matrix}$

and may compute the ith component of the center of the class centroidsas

$\begin{matrix}{{\overset{\_}{x}}_{i}^{c} = {\sum\limits_{k = 1}^{K}\;{\frac{{\overset{\_}{x}}_{ik}}{K}.}}} & (2)\end{matrix}$

The classification engine 114 may also calculate the ith component ofthe overall centroid as:

$\begin{matrix}{{\overset{\_}{x}}_{i} = {\sum\limits_{j = 1}^{n}\;{\frac{x_{ij}}{n}.}}} & (3)\end{matrix}$

The classification engine 114 may then perform a transformation thatincludes adjusting the ith entry in each element of the training dataset T0.train 304 by adding the difference given by:Δ=− x _(i) ^(c).  (4)

In some implementations, the transformation performed at step 320includes a shift other than the one described above with reference toEqs. 1-4, a rotation, a shear, a combination of these transformations,or any other linear or non-linear transformation.

At step 324, the classification engine 114 transforms the test data setT0.test 308. The transformation applied to the test data set T0.test308, correctedData, is the same type of transformation applied to thetraining data set T0.train 304 at step 320, but applied with respect tothe arguments T0.test 308 and predc1.test 318 instead of T0.train 304and predc1.train 314. For example, if the elements of the training dataset T0.train 304 are adjusted at step 320 by the value of Δ given by Eq.4 as calculated with respect to the centroids of the classes of thetraining data set T0.train 304, then the elements of the test data setT0.test 308 are adjusted at step 324 by the value of Δ given by Eq. 4 ascalculated with respect to the centroids of the classes of the test dataset T0.test 308. The result of the transformation of step 324 is atransformed test data set, T0.test.2 326, which the classificationengine 114 may store in a memory.

At step 327, the classification engine 114 determines whether the valueof the iteration counter i is equal to 1. If so, the classificationengine 114 proceeds to execute step 328, in which the classificationengine 114 uses the transformed training data set T0.train.2 322 and thetraining class set c1.train 306 to generate a second classifier rf2 329.As described above with reference to Step 332, and to step 336, anymachine-learning technique may be applied to generate the classifier atstep 328. The second classifier rf2 329 may be of the same type as thefirst classifier rf 314 (e.g., both SVM classifiers), or of a differenttype.

At step 331, the classification engine 114 increments the iterationcounter i, then proceeds to execute step 333, in which theclassification engine 114 applies the second classifier rf2 329 to thetransformed test data set T0.test.2 326 (as generated by theclassification engine 114 at step 324). The output of step 333 is a setof predicted classifications predc1.test.2 330 for the transformed dataset T0.test.2 326. The classification engine 114 may output thepredicted classifications to a display device, a printing device, astoring device, another device in communication with the classificationengine 114 across a network or any other device internal or external tothe system 100.

At step 332, the classification engine 114 determines whether there areany differences between the classifications of the predictedclassification set predc1.test 318 (as generated at step 316) and thepredicted classifications set predc1.test.2 330 (as generated at step328). If the sets of predicted classifications agree (i.e., for eachdata sample in the test data set T0.test 308, the predicted class forthat data sample is the same between the two predicted classificationsset), then the classification engine 114 proceeds to step 338 andoutputs the predicted classification set predc1.test.2 330(equivalently, the predicted classification set predc1.test 318) as thefinal classification of the test data set T0.test 308.

If the classification engine 114 identifies differences between theclassification data set predc1.test 318 and the classification data setpredc1.test.2 330, the classification engine 114 proceeds to step 334and replaces the previously stored value of the test data set T0.test308 with the value of the transformed test data set T0.test.2 326 (asgenerated by the transformation of step 324). As a result, the test dataset T0.test 308 has the values of the transformed test data setT0.test.2 326. The classification engine 114 proceeds to step 336 andreplaces the previously stored value of the predicted classification setpredc1.test 318 (as generated at step 316) with the value of thepredicted classification set predc1.test.2 330 (as generated at step328). As a result, the predicted classification set predc1.test 318 hasthe values of the predicted classification set predc1.test.2 330.

Once the value of the test data set T0.test 308 has been updated withthe value of the transformed test data set T0.test.2 326 and thepredicted classification set predc1.test 318 has been updated with thevalues of the predicted classification set predc1.test.2 330, theclassification engine 114 returns to step 324 to perform a newtransformation and iterates this process until the classification engine114 determines that there is no difference between the predictedclassifications (at step 332).

The classifier performance monitoring engine 116 may analyze theperformance of the final classification produced by the classificationengine 114 at the conclusion of the process of FIG. 3 using a suitableperformance metric. In certain embodiments, the performance metric mayinclude an error rate. The performance metric may also include thenumber of correct predictions divided by the total predictionsattempted. The performance metric may be any suitable measure withoutdeparting from the scope of the present disclosure.

Implementations of the present subject matter can include, but are notlimited to, systems methods and computer program products comprising oneor more features as described herein as well as articles that comprise amachine-readable medium operable to cause one or more machines (e.g.,computers, robots) to result in operations described herein. The methodsdescribed herein can be implemented by one or more processors or enginesresiding in a single computing system or multiple computing systems.Such multiple computing systems can be connected and can exchange dataand/or commands or other instructions or the like via one or moreconnections, including but not limited to a connection over a network(e.g. the Internet, a wireless wide area network, a local area network,a wide area network, a wired network, or the like), via a directconnection between one or more of the multiple computing systems.

FIG. 4 is a block diagram of a computing device, such as any of thecomponents of system 100 of FIG. 1 including circuitry for performingprocesses described with reference to FIGS. 1-3. Each of the componentsof system 100 may be implemented on one or more computing devices 400.In certain aspects, a plurality of the above-components and databasesmay be included within one computing device 400. In certainimplementations, a component and a database may be implemented acrossseveral computing devices 400.

The computing device 400 comprises at least one communications interfaceunit, an input/output controller 410, system memory, and one or moredata storage devices. The system memory includes at least one randomaccess memory (RAM 402) and at least one read-only memory (ROM 404). Allof these elements are in communication with a central processing unit(CPU 406) to facilitate the operation of the computing device 400. Thecomputing device 400 may be configured in many different ways. Forexample, the computing device 400 may be a conventional standalonecomputer or alternatively, the functions of computing device 400 may bedistributed across multiple computer systems and architectures. Thecomputing device 400 may be configured to perform some or all ofdata-splitting, differentiating, classifying, scoring, ranking andstoring operations. In FIG. 4, the computing device 400 is linked, vianetwork or local network, to other servers or systems.

The computing device 400 may be configured in a distributedarchitecture, wherein databases and processors are housed in separateunits or locations. Some such units perform primary processing functionsand contain at a minimum a general controller or a processor and asystem memory. In such an aspect, each of these units is attached viathe communications interface unit 408 to a communications hub or port(not shown) that serves as a primary communication link with otherservers, client or user computers and other related devices. Thecommunications hub or port may have minimal processing capabilityitself, serving primarily as a communications router. A variety ofcommunications protocols may be part of the system, including, but notlimited to: Ethernet, SAP, SAS™, ATP, BLUETOOTH™, GSM and TCP/IP.

The CPU 406 comprises a processor, such as one or more conventionalmicroprocessors and one or more supplementary co-processors such as mathco-processors for offloading workload from the CPU 406. The CPU 406 isin communication with the communications interface unit 408 and theinput/output controller 410, through which the CPU 406 communicates withother devices such as other servers, user terminals, or devices. Thecommunications interface unit 408 and the input/output controller 410may include multiple communication channels for simultaneouscommunication with, for example, other processors, servers or clientterminals. Devices in communication with each other need not becontinually transmitting to each other. On the contrary, such devicesneed only transmit to each other as necessary, may actually refrain fromexchanging data most of the time, and may require several steps to beperformed to establish a communication link between the devices.

The CPU 406 is also in communication with the data storage device. Thedata storage device may comprise an appropriate combination of magnetic,optical or semiconductor memory, and may include, for example, RAM 402,ROM 404, flash drive, an optical disc such as a compact disc or a harddisk or drive. The CPU 406 and the data storage device each may be, forexample, located entirely within a single computer or other computingdevice; or connected to each other by a communication medium, such as aUSB port, serial port cable, a coaxial cable, an Ethernet type cable, atelephone line, a radio frequency transceiver or other similar wirelessor wired medium or combination of the foregoing. For example, the CPU406 may be connected to the data storage device via the communicationsinterface unit 408. The CPU 406 may be configured to perform one or moreparticular processing functions.

The data storage device may store, for example, (i) an operating system412 for the computing device 400; (ii) one or more applications 414(e.g., computer program code or a computer program product) adapted todirect the CPU 406 in accordance with the systems and methods describedhere, and particularly in accordance with the processes described indetail with regard to the CPU 406; or (iii) database(s) 416 adapted tostore information that may be utilized to store information required bythe program. In some aspects, the database(s) includes a databasestoring experimental data, and published literature models.

The operating system 412 and applications 414 may be stored, forexample, in a compressed, an uncompiled and an encrypted format, and mayinclude computer program code. The instructions of the program may beread into a main memory of the processor from a computer-readable mediumother than the data storage device, such as from the ROM 404 or from theRAM 402. While execution of sequences of instructions in the programcauses the CPU 406 to perform the process steps described herein,hard-wired circuitry may be used in place of, or in combination with,software instructions for implementation of the processes of the presentinvention. Thus, the systems and methods described are not limited toany specific combination of hardware and software.

Suitable computer program code may be provided for performing one ormore functions in relation to modeling, scoring and aggregating asdescribed herein. The program also may include program elements such asan operating system 412, a database management system and “devicedrivers” that allow the processor to interface with computer peripheraldevices (e.g., a video display, a keyboard, a computer mouse, etc.) viathe input/output controller 410.

A computer program product comprising computer-readable instructions isalso provided. The computer-readable instructions, when loaded andexecuted on a computer system, cause the computer system to operateaccording to the method, or one or more steps of the method describedabove. The term “computer-readable medium” as used herein refers to anynon-transitory medium that provides or participates in providinginstructions to the processor of the computing device 400 (or any otherprocessor of a device described herein) for execution. Such a medium maytake many forms, including but not limited to, non-volatile media andvolatile media. Non-volatile media include, for example, optical,magnetic, or opto-magnetic disks, or integrated circuit memory, such asflash memory. Volatile media include dynamic random access memory(DRAM), which typically constitutes the main memory. Common forms ofcomputer-readable media include, for example, a floppy disk, a flexibledisk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM,DVD, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, a RAM, a PROM, an EPROM orEEPROM (electronically erasable programmable read-only memory), aFLASH-EEPROM, any other memory chip or cartridge, or any othernon-transitory medium from which a computer can read.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the CPU 406 (or anyother processor of a device described herein) for execution. Forexample, the instructions may initially be borne on a magnetic disk of aremote computer (not shown). The remote computer can load theinstructions into its dynamic memory and send the instructions over anEthernet connection, cable line, or even telephone line using a modem. Acommunications device local to a computing device 400 (e.g., a server)can receive the data on the respective communications line and place thedata on a system bus for the processor. The system bus carries the datato main memory, from which the processor retrieves and executes theinstructions. The instructions received by main memory may optionally bestored in memory either before or after execution by the processor. Inaddition, instructions may be received via a communication port aselectrical, electromagnetic or optical signals, which are exemplaryforms of wireless communications or data streams that carry varioustypes of information.

EXAMPLE

The following public datasets are downloaded from the Gene ExpressionOmnibus (GEO) (http://www.ncbi.nlm.nih.gov/geo/) repository:

-   -   a. GSE10106        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE10106)    -   b. GSE10135        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE10135)    -   c. GSE11906        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE11906)    -   d. GSE11952        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE11952)    -   e. GSE13933        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE13933)    -   f. GSE19407        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE19407)    -   g. GSE19667        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE19667)    -   h. GSE20257        (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE20257)    -   i. GSE5058 (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE5058)    -   j. GSE7832 (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE7832)    -   k. GSE8545 (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE8545).

The training datasets are on the Affymetrix platform (HGU-133+2). Rawdata files are read by the ReadAffy function of the affy package(Gautier, 2004) belonging to Bioconductor (Gentleman, 2004) in R (RDevelopment Core Team, 2007), and the quality is controlled by:generating RNA degradation plots (with the AffyRNAdeg function of theaffy package), NUSE and RLE plots (with the function affyPLM(Brettschneider, 2008)), and calculating the MA(RLE) values; excludingarrays from the training datasets that fell below a set of thresholds onthe quality control checks or that are duplicated in the above datasets;and normalizing arrays that pass quality control checks using the gcrmaalgorithm (Wu, 2004). Training set sample classifications are obtainedfrom the series matrix file of the GEO database for each dataset. Theoutput consists of a gene expression matrix with 54675 probesets for 233samples (28 COPD samples and 205 control sample). To make a balanceddata set, the COPD samples were multiple time to obtain 224 COPD samplesbefore the Duel Ensemble method as described in copending U.S.provisional application 61/662,812 is applied. With a combined data setwhich contains 205 control and 224 COPD patients, a gene signature with409 genes was built. 850 binary values were used in the random vectors.The classification methods used in the method included the following Rpackages: lda, svm, randomForest, knn, pls.lda and pamr. Maximumiteration was set to be 5000. The Matthew's Correlation Coefficient(MCC) and accuracy in cross validation process in training data set is0.743 and 0.87 respectively. The heatmap of the gene signature intraining data set is shown in FIG. 5. In the heatmap of FIG. 5, the geneexpression value was centered by row. The colors of the heatmap may notbe clearly shown in grey scale, but the data of FIG. 5 show that controldata are shown on the left, and COPD data are shown on the right. Thetest data set is an unpublished data set obtained from a commercialsupplier (Genelogic), which contains 16 control samples and 24 COPDsamples. Without applying the transformation invariant method of theinvention, the gene signature generated by Dual Ensemble correctlypredicted 29 samples out of total 40 samples. The accuracy is 0.725, andthe MCC is 0.527. In the 16 control samples, the gene signaturecorrectly predicted 15 as control but erroneously predicted 1 as COPD.Among the 24 COPD samples, the gene signature correctly predicted 14 asCOPD samples but erroneously predicted 10 as control.

However, when the transformation invariant method was applied with ashift according to the center of two or multiple classes and a maximumiterations set to 100. The same gene signature correctly predicted 30samples out of total 40 samples. The accuracy is 0.75, and the MCC is0.533. In the 16 control samples, the gene signature correctly predicted14 as control but erroneously predicted 2 as COPD. Among the 24 COPDsamples, the gene signature correctly predicted 16 as COPD samples buterroneously predicted 8 as control.

While implementations of the invention have been particularly shown anddescribed with reference to specific examples, it should be understoodby those skilled in the art that various changes in form and detail maybe made therein without departing from the spirit and scope of thedisclosure.

The invention claimed is:
 1. A computer-implemented method ofclassifying a data set into two or more classes, comprising: (a)receiving, by a biomarker generator, a training data set and a trainingclass set, the training class set including a set of known labels, eachknown label identifying a class associated with each element in thetraining data set; (b) receiving, by the biomarker generator, a testdata set; (c) generating, by the biomarker generator, a first classifierfor the training data set by applying a first machine learning techniqueto the training data set and the training class set; (d) generating, bythe biomarker generator, a first test class set by classifying theelements in the test data set according to the first classifier; (e)transforming, by the biomarker generator, the training data set byshifting the elements in the training data set by an amountcorresponding to a center of a set of training class centroids, whereineach training class centroid is representative of a center of a subsetof elements in the training data set; (f) for each of a plurality ofiterations: (i) transforming, by the biomarker generator, the test dataset by shifting the elements in the test data set by an amountcorresponding to a center of a set of test class centroids, wherein eachtest class centroid is representative of a center of a subset ofelements in the test data set; (ii) generating, by the biomarkergenerator, a second test class set by classifying the elements in thetransformed test data set according to a second classifier, wherein thesecond classifier is generated by applying a second machine learningtechnique to the transformed training data set and the training classset; and (iii) storing, by the biomarker generator, when the first testclass set and the second test class set differ, the second test classset as the first test class set and the transformed test data set as thetest data set and returning to step (i); and (g) outputting, by thebiomarker generator, when the first test class set is the same as thesecond test class set, the second test class set.
 2. The method of claim1, wherein the elements of the training data set represent geneexpression data for a patient with a disease, for a patient resistant tothe disease, or for a patient without the disease.
 3. The method ofclaim 1, wherein the training data set is formed from a random subset ofsamples in an aggregate data set, and the test data set is formed from aremaining subset of samples in the aggregate data set.
 4. The method ofclaim 1, wherein: the test data set includes a rest set of known labels,each known label identifying a class associated with each element in thetest data set; the first test class set includes a set of predictedlabels for the test data set; and the second test class set includes aset of predicted labels for the transformed test data set.
 5. The methodof claim 1, wherein the shifting at step (e) includes applying arotation, a linear transformation, or a non-linear transformation to thetraining data set to obtain the transformed training data set.
 6. Themethod of claim 1, wherein the shifting at step (i) includes applying arotation, a shear, a linear transformation, or a nonlineartransformation to the test data set to obtain the transformed test dataset.
 7. The method of claim 1, further comprising: (h) comparing, by thebiomarker generator, the first test class set to the second test classset for each of the plurality of iterations.
 8. The method of claim 1,wherein the transforming at step (e) is performed by applying the sametransformation of step (i).
 9. The method of claim 1, furthercomprising: (h) providing, by the biomarker generator, the second testclass set to a display device, a printing device, or a storing device.10. The method of claim 1, wherein the first test class set and thesecond test class set differ if any element of the first test class setdiffers from a corresponding element of the second test class set. 11.The method of claim 1, wherein the second test class set includes a setof predicted labels for the transformed test data set, the methodfurther comprising: evaluating, by the biomarker generator, the secondclassifier by computing a performance metric representative of a numberof correct predicted labels in the second test class set divided by atotal number of predicted labels.
 12. The method of claim 1, wherein thefirst machine learning technique and the second machine learningtechnique are the same.
 13. The method of claim 1, further comprising:(h) determining, by the biomarker generator, based on the second classset, a set of candidate biomarkers and at least one candidate errorrate.
 14. The method of claim 13, further comprising: (i) receiving, bya biomarker consolidator in communication with the biomarker generator,the set of candidate biomarkers and the candidate error rate; (j)determining, by the biomarker generator, a performance measure and sizeof each set of candidate biomarkers; and (k) selecting, by the biomarkergenerator, based on the performance measure and size of each set ofcandidate biomarkers, an optimal biomarker.
 15. The method of claim 13,further comprising: (l) controlling, by a central control unit incommunication with the biomarker generator and the biomarkerconsolidator, at least partially the operation of the biomarkergenerator and the biomarker consolidator.
 16. A non-transitorycomputer-readable storage medium having instructions stored thereonthat, when executed in a computerized system comprising at least oneprocessor, cause said at least one processor to carry out operations,the operations comprising: (a) receiving a training data set and atraining class set, the training class set including a set of knownlabels, each known label identifying a class associated with eachelement in the training data set; (b) receiving a test data set; (c)generating a first classifier for the training data set by applying afirst machine learning technique to the training data set and thetraining class set; (d) generating a first test class set by classifyingthe elements in the test data set according to the first classifier; (e)transforming the training data set by shifting the elements in thetraining data set by an amount corresponding to a center of a set oftraining class centroids, wherein each training class centroid isrepresentative of a center of a subset of elements in the training dataset; (f) for each of a plurality of iterations: (i) transforming thetest data set by shifting the elements in the test data set by an amountcorresponding to a center of a set of test class centroids, wherein eachtest class centroid is representative of a center of a subset ofelements in the test data set (ii) generating a second test class set byclassifying the elements in the transformed test data set according to asecond classifier, wherein the second classifier is generated byapplying a second machine learning technique to the transformed trainingdata set and the training class set; and (iii) when the first test classset and the second test class set differ, storing the second test classset as the first test class set, storing the transformed test data setas the test data set, and returning to step (i); and (g) outputting,when the first test class set is the same as the second test class set,the second test class set.
 17. The non-transitory computer-readablestorage medium of claim 16, the operations further comprising: (h)outputting, when the first test class set and the second test class setdo not differ, the second test class set.
 18. The non-transitorycomputer-readable storage medium of claim 16, wherein the elements ofthe training data set represent gene expression data for a patient witha disease, for a patient resistant to the disease, or for a patientwithout the disease.
 19. A computerized system comprising: a biomarkergenerator; and a memory coupled to the biomarker generator and havinginstructions stored thereon that, when executed, cause the biomarkergenerator to perform operations, the operations comprising: (a)receiving a training data set and a training class set, the trainingclass set including a set of known labels, each known label identifyinga class associated with each element in the training data set; (b)receiving a test data set; (c) generating a first classifier for thetraining data set by applying a first machine learning technique to thetraining data set and the training class set; (d) generating a firsttest class set by classifying the elements in the test data setaccording to the first classifier; (e) transforming the training dataset by shifting the elements in the training data set by an amountcorresponding to a center of a set of training class centroids, whereineach training class centroid is representative of a center of a subsetof elements in the training data set; (f) for each of a plurality ofiterations: (i) transforming the test data set by shifting the elementsin the test data set by an amount corresponding to a center of a set oftest class centroids, wherein each test class centroid is representativeof a center of a subset of elements in the test data set; (ii)generating a second test class set by classifying the elements in thetransformed test data set according to a second classifier, wherein thesecond classifier is generated by applying a second machine learningtechnique to the transformed training data set and the training classset; and (iii) storing, when the first test class set and the secondtest class set differ, the second test class set as the first test classset and the transformed test data set as the test data set and returningto step (i); and (g) outputting, when the first test class set is thesame as the second test class set, the second test class set.
 20. Thecomputerized system of claim 19, the operations further comprising: (h)outputting, when the first test class set and the second test class setdo not differ, the second test class set.
 21. The computerized system ofclaim 19, wherein the elements of the training data set represent geneexpression data for a patient with a disease, for a patient resistant tothe disease, or for a patient without the disease.
 22. The computerizedsystem of claim 19, wherein the training data set is formed from arandom subset of samples in an aggregate data set, and the test data setis formed from a remaining subset of samples in the aggregate data set.